Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci(MM-63199): code coverage tracking #8642

Draft
wants to merge 75 commits into
base: main
Choose a base branch
from
Draft

ci(MM-63199): code coverage tracking #8642

wants to merge 75 commits into from
+244 −1

Conversation

rahimrahman
Copy link
Contributor

@rahimrahman rahimrahman commented Feb 28, 2025 •
edited
Loading

Summary

Introducing Code Coverage Report. Every time a PR is being merge to main, it will run test:coverage and save the code coverage summary as an artifact. At the same time, a github action cache will store the last run id. When a PR is created and anytime a new commit is pushed to the PR, the action will pull coverage summary from main (which is stored as an artifact), and then run the test-coverage on current code. Using a shell script, the two code coverage summaries will be compared in order to produce the report.

Currently, it has a 1% threshold, which can be modified in the compare-coverage.sh. So, if the current code changes went down by 1% (for either lines, statement, functions or branch), it will cause the action to post a warning along with the coverage report summary.

Currently, if the test fail, it will trigger a failure (this is the same as today). But if the coverage report dips below the threshold, then this should not fail the PR.

This can be improved in so many ways:

  • code coverage is showing to shift even when doing non JS/TS code changes
  • getting coverage report summary every commit can be annoying?

Ticket Link

https://mattermost.atlassian.net/browse/MM-63199

Checklist

  • Added or updated unit tests (required for all new features)
  • Has UI changes
  • Includes text changes and localization file updates
  • Have tested against the 5 core themes to ensure consistency between them.
  • Have run E2E tests by adding label E2E iOS tests for PR.

Device Information

This PR was tested on:

Screenshots

Release Note

@enahum
Copy link
Contributor

enahum commented Mar 10, 2025

Coverage Comparison Report

```+-----------------+------------+------------+-----------+| Metric | Current | Previous | Diff |+-----------------+------------+------------+-----------+| Lines | 85.25% | 85.25% | 0.00% || Statements | 85.13% | 85.13% | 0.00% || Branches | 70.81% | 70.83% | -0.02% || Functions | 83.22% | 83.22% | 0.00% |+-----------------+------------+------------+-----------+\n```

@enahum
Copy link
Contributor

enahum commented Mar 10, 2025

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric         | Current    | Previous   | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |      85.25% |      85.25% |      0.00% |
| Statements      |      85.13% |      85.13% |      0.00% |
| Branches        |      70.81% |      70.81% |      0.00% |
| Functions       |      83.22% |      83.22% |      0.00% |
+-----------------+------------+------------+-----------+

@enahum
Copy link
Contributor

enahum commented Mar 10, 2025

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric         | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |      85.25% |      85.25% |      0.00% |
| Statements      |      85.13% |      85.13% |      0.00% |
| Branches        |      70.81% |      70.83% |      0.02% |
| Functions       |      83.22% |      83.22% |      0.00% |
+-----------------+------------+------------+-----------+

@enahum
Copy link
Contributor

enahum commented Mar 12, 2025

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric          | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |     85.22% |     85.25% |     0.03% |
| Statements      |     85.10% |     85.13% |     0.03% |
| Branches        |     70.72% |     70.76% |     0.04% |
| Functions       |     83.28% |     83.36% |     0.08% |
+-----------------+------------+------------+-----------+
| Total           |     81.08% |     81.12% |     0.04% |
+-----------------+------------+------------+-----------+

@enahum
Copy link
Contributor

enahum commented Mar 12, 2025

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric          | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |     85.22% |     85.25% |     0.03% |
| Statements      |     85.10% |     85.13% |     0.03% |
| Branches        |     70.72% |     70.76% |     0.04% |
| Functions       |     83.28% |     83.36% |     0.08% |
+-----------------+------------+------------+-----------+
| Total           |     81.08% |     81.12% |     0.04% |
+-----------------+------------+------------+-----------+

@github-actions GitHub Actions
Copy link

Hello world ! 👋

@github-actions GitHub Actions
Copy link

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric          | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |     85.22% |     85.25% |     0.03% |
| Statements      |     85.10% |     85.13% |     0.03% |
| Branches        |     70.72% |     70.76% |     0.04% |
| Functions       |     83.28% |     83.36% |     0.08% |
+-----------------+------------+------------+-----------+
| Total           |     81.08% |     81.12% |     0.04% |
+-----------------+------------+------------+-----------+

@github-actions GitHub Actions
Copy link

::error::Lines coverage has decreased by more than 1.0% (.03%)
::error::Statements coverage has decreased by more than 1.0% (.03%)
::error::Branches coverage has decreased by more than 1.0% (.03%)
::error::Functions coverage has decreased by more than 1.0% (.08%)

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric          | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |     85.22% |     85.25% |     0.03% |
| Statements      |     85.10% |     85.13% |     0.03% |
| Branches        |     70.72% |     70.75% |     0.03% |
| Functions       |     83.28% |     83.36% |     0.08% |
+-----------------+------------+------------+-----------+
| Total           |     81.08% |     81.12% |     0.04% |
+-----------------+------------+------------+-----------+

⚠️ Warning: One or more coverage metrics have decreased by more than 1.0%

@github-actions GitHub Actions
Copy link

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric          | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |     85.22% |     85.25% |     0.03% |
| Statements      |     85.10% |     85.13% |     0.03% |
| Branches        |     70.72% |     70.76% |     0.04% |
| Functions       |     83.28% |     83.36% |     0.08% |
+-----------------+------------+------------+-----------+
| Total           |     81.08% |     81.12% |     0.04% |
+-----------------+------------+------------+-----------+

⚠️ Warning: One or more coverage metrics have decreased by more than 1.0%

@github-actions GitHub Actions
Copy link

Coverage Comparison Report

+-----------------+------------+------------+-----------+
| Metric          | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |     85.22% |     85.25% |     0.03% |
| Statements      |     85.10% |     85.13% |     0.03% |
| Branches        |     70.72% |     70.76% |     0.04% |
| Functions       |     83.28% |     83.36% |     0.08% |
+-----------------+------------+------------+-----------+
| Total           |     81.08% |     81.12% |     0.04% |
+-----------------+------------+------------+-----------+

@enahum
Copy link
Contributor

enahum commented Mar 12, 2025 •
edited by github-actions bot
Loading

Coverage Comparison Report

Generated March 13, 2025 at 14:57:47 UTC

+-----------------+------------+------------+-----------+
| Metric          | Main       | This PR    | Diff      |
+-----------------+------------+------------+-----------+
| Lines           |     85.25% |     85.20% |    -0.05% |
| Statements      |     85.13% |     85.08% |    -0.05% |
| Branches        |     70.76% |     70.71% |    -0.05% |
| Functions       |     83.36% |     83.25% |    -0.11% |
+-----------------+------------+------------+-----------+
| Total           |     81.12% |     81.06% |    -0.06% |
+-----------------+------------+------------+-----------+

⚠️ Warning: One or more coverage metrics have decreased by more than 0.5%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Willyfrog Willyfrog Willyfrog approved these changes

Assignees
No one assigned
Labels
release-note
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@rahimrahman @enahum @Willyfrog @mm-cloud-bot